K-Space at TRECVID 2007

In this paper we describe K-Space participation in TRECVid 2007. K-Space participated in two tasks, highlevel feature extraction and interactive search. We present our approaches for each of these activities and provide a brief analysis of our results. Our high-level feature submission utilized multi-modal low-level features which included visual, audio and temporal elements. Specific concept detectors (such as Face detectors) developed by K-Space partners were also used. We experimented with different machine learning approaches including logistic regression and support vector machines (SVM). Finally we also experimented with both early and late fusion for feature combination. This year we also participated in interactive search, submitting 6 runs. We developed two interfaces which both utilized the same retrieval functionality. Our objective was to measure the effect of context, which was supported to different degrees in each interface, on user performance. The first of the two systems was a ‘shot’ based interface, where the results from a query were presented as a ranked list of shots. The second interface was ‘broadcast’ based, where results were presented as a ranked list of broadcasts. Both systems made use of the outputs of our high-level feature submission as well as low-level visual features. 1 Overview of K-Space K-Space is a European Network of Excellence (NoE) in semantic inference for semi-automatic annotation and retrieval of multimedia content [1] which is in the second year of its three year funding. It is coordinated by Queen Mary University of London (QMUL) and the partner responsible for coordinating the K-Space participation in TRECVid is Dublin City University. K-Space is focused on the research and convergence of three themes: content-based multimedia analysis, knowledge extraction and semantic multimedia. This paper describes the K-Space participation in TRECVid 2007. TRECVid ([39]) is an annual benchmarking evaluation campaign for research groups to use common data and queries to assess the relative performance of their techniques in an open, metrics-based forum. 2007 marks the 7th year of TRECVid. 2 Audio-Visual Features The K-Space submission in both feature detection and interactive search made use of several feature detectors developed by K-Space partners in prior work. Later in this section we outline the specific concept detectors contributed by individual K-Space partners used during KSpace participation in TRECVid 2007. First, though, we present some detail on the low-level visual features used. For the remainder of this paper, the term ‘concept’ will refer to a high-level feature (e.g. ‘Car’). As no common keyframe set was released as part of the TRECVid 2007 collection, we extracted our own set of keyframes. Our keyframe selection strategy was to extract every second I-Frame from each shot. This gives us far more keyframes than the usual one-keyframe-per-shot which has been the norm in previous TRECVids and in fact gives us about 1 keyframe per second of video. For the remainder of this paper, we will refer to these images as K-Frames. We extracted low-level visual features from K-frames using several feature descriptors based on the MPEG-7 XM. These descriptors were implemented as part of the aceToolbox, a toolbox of low-level audio and visual analysis tools developed as part of our participation in the EU aceMedia project [2]. We made use of six different global visual descriptors. These descriptors were Colour Layout, Colour Moments, Colour Structure, Homogenous Texture, Edge Histogram and Scalable Colour. A complete description of each of these descriptors can be found in [24]. We also segmented each of the K-frames into regions. We considered several approaches to image segmentation [3], [18], [7], [23], when selecting the method for automatically partitioning K-frames into large regions which reflect the objects (or their parts) present in the image. We considered not only the accuracy of segmented regions in terms of how well they mapped to object, but also the typical number of regions produced by a given algorithm and its computational cost to execute. The number of regions was a particularly relevant factor since large regions are typically more suited to subsequent robust feature estimation. We decided to use the approach proposed in [3] which is based on the well known Recursive Shortest Spanning Tree (RSST) method utilizing the more perceptually uniform L*U*V* color model and syntactic visual features to improve the quality of the segmentation. The syntactic features represent geometric properties of regions and their spatial configurations. This approach allowed satisfactory segmentation of various types of scenes into a set of large and typically meaningful regions without adjustment to algorithm parameters. This set of K-frames and their features and regions were distributed to all K-Space TRECVid partners so that each could run their own feature detectors on the video and send the output back to DCU for coordination. The remainder of this section describes the partner contributions. Proc. TRECVID 2007 Workshop, November 2007, Gaithersburg, MD, USA.